Hey @eschnett
Given that the approach via libcxxwrap_julia basically takes a detour via a low-level (Julia) API anyway, I think that it definitely makes sense to go via C instead, giving a much more stable and ubiquitous low-level API for bindings. I'll take a closer look at this once I find the time.

(This is responding to #1025 (comment) as well)

A high-level design question for a C interface is how dynamically allocated structures or arrays should be handled. I am currently using this approach:

    char* get_stuff();
    size_t get_stuffSize();

where get_stuff uses malloc to allocate memory, and get_stuffSize return the size of the allocated memory (if necessary, not necessary e.g. for strings which are NUL-terminated.) The caller needs to free the returned memory.

Another approach that could be used would be

    size_t get_stuff(char* ptr, size_t provided_size);

where the caller needs to allocate the memory. get_stuff returns the actually needed size, and writes things to ptr only if ptr is non-NULL, and if the provided_size is large enough. A variant of that is

    int get_suff(char *ptr, size_t *size);

which writes the necessary size into *size (if non-null), and writes the data into *ptr (if non-null), returning either nothing or an error code. In both cases the caller is responsible for both allocating and freeing the memory.

This choice is on one hand inconsequential – it just specifies how memory is allocated or freed – but is on the other hand pervasive throughout the API.

A high-level design question for a C interface is how dynamically allocated structures or arrays should be handled. I am currently using this approach:

    char* get_stuff();
    size_t get_stuffSize();

where get_stuff uses malloc to allocate memory, and get_stuffSize return the size of the allocated memory (if necessary, not necessary e.g. for strings which are NUL-terminated.) The caller needs to free the returned memory.

Another approach that could be used would be

    size_t get_stuff(char* ptr, size_t provided_size);

where the caller needs to allocate the memory. get_stuff returns the actually needed size, and writes things to ptr only if ptr is non-NULL, and if the provided_size is large enough. A variant of that is

    int get_suff(char *ptr, size_t *size);

which writes the necessary size into *size (if non-null), and writes the data into *ptr (if non-null), returning either nothing or an error code. In both cases the caller is responsible for both allocating and freeing the memory.

This choice is on one hand inconsequential – it just specifies how memory is allocated or freed – but is on the other hand pervasive throughout the API.

I think that your current approach is fine. openPMD-api is not so low-level that memory management needs to be done by the user IMO. I'd suggest that we add either something like void openPMD_free(void *) or even structure-specific openPMD_<structname>_free(...) functions, so we can avoid future breaking changes.

Is this PR ready for review? Give me a note if you need help with anything (like CMake integration of the tests).

@franzpoeschel The code is not yet ready for review, some functions are still missing.